lib: improve transactions auto-cleanup logic

lib: improve transactions auto-cleanup logic

This fixes some aspects of OstreeRepoAutoTransaction and re-aligns
it with the logic in flatpak. Specifically:
 * link to the underlying repo through refcounting
 * bridge internal errors to warning messages
 * verify the input pointer type

This is a preparation step before exposing this logic as a public API.